ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

Abstract Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free that directly raw text (bytes characters) have many benefits: They can process in any out the box, they are more robust noise, and minimize technical debt by removing complex error-prone preprocessing pipelines. Because byte character longer than token sequences, past work has often introduced new model architectures designed amortize cost operating text. In this paper, we show a standard Transformer architecture be with minimal modifications sequences. We characterize trade-offs terms parameter count, training FLOPs, inference speed, byte-level competitive their token-level counterparts. also demonstrate significantly noise perform better tasks sensitive spelling pronunciation. As part our contribution, release set based T5 architecture, as well all code data experiments.1